Repository navigation
fix: preserve prepared ROUND and TRUNCATE value domains - #29508
Conversation
Qodo reviews are paused for this user.Troubleshooting steps vary by plan Learn more → On a Teams plan? Using GitHub Enterprise Server, GitLab Self-Managed, or Bitbucket Data Center? |
| // normal comparison domain. | ||
| func preparedZeroPrecisionRoundParam(ctx context.Context, expr *Expr) *Expr { | ||
| fn := expr.GetF() | ||
| if fn == nil || fn.Func == nil || (fn.Func.GetObjName() != "round" && fn.Func.GetObjName() != "truncate") || len(fn.Args) != 2 { |
There was a problem hiding this comment.
ROUND(expr) is also a valid one-argument form with default precision 0, but this proof only accepts two arguments. Consequently ROUND(?) and ROUND((SELECT ?)) skip the safe unique-domain rewrite. In the local 100k-row decimal-table check with bound value '99999.0', these forms read 13 blocks / 100,000 input rows, while the explicit ROUND(?,0) forms pruned to 1 block / 1 row. Please consider treating an omitted precision as literal 0 and covering both direct and scalar-subquery forms in a follow-up. This is a missed optimization; results remained correct.
| return nil | ||
| } | ||
| fn := current.GetF() | ||
| if fn == nil || fn.Func == nil || fn.Func.GetObjName() != "=" || len(fn.Args) != 2 { |
There was a problem hiding this comment.
This scalar-subquery rewrite is gated to =, so safe range predicates such as id >= ROUND((SELECT ?),0) keep the cast on the indexed column and miss block pruning. In the local check, that form read 13 blocks / 100,000 rows, while the direct-parameter id >= ROUND(?,0) form read 1 block. A follow-up can apply the same proof to supported comparison operators and bind the rewritten expression using the original operator name instead of hard-coding = here. Results remained correct in this case.
What type of PR is this?
Which issue(s) this PR fixes:
Closes #29505
Closes #29509
Closes #29471
Closes #29506
Closes #29511
Closes #29512
Closes #29510
Closes #29514
Closes #29515
Closes #29516
Closes #29517
What this PR does / why we need it:
Keep the prepared
ROUND/TRUNCATEvalue argument's numeric domain open until EXECUTE, while the precision argument retains its integer domain. Rebuild cached expressions with all arguments and preserve explicit CAST and set-operation output domains ([Bug]: Prepared ROUND/TRUNCATE lose runtime numeric domain #29505).Keep the prepared generation and snapshot binding when
EXPLAIN ANALYZE FORCE EXECUTEcompiles the executable plan ([Bug]: EXPLAIN ANALYZE FORCE EXECUTE fails on prepared plan snapshot binding #29509).Serialize JSON-to-CHAR expression casts with JSON string quotes while preserving existing assignment behavior ([Compatibility]: JSON-to-character CAST unquotes JSON strings instead of serializing them #29471).
For prepared integer-key comparisons, prove that an integral text/float binding can use the native integer domain without changing existing DOUBLE comparison results; preserve fallback at and beyond the signed 2^53 collision boundary, for fractional/malformed/out-of-range values, and for unsigned edge cases. Carry the existing prepared diagnostic proof into executable EXPLAIN so its actual scan uses the block filter ([Bug]: Prepared non-integer values on signed BIGINT keys lose block filtering #29506).
For SELECT comparisons between an integer key and a wider or differently signed integer binding, narrow only a proven safe runtime value to the key domain. Preserve the existing fallback for out-of-range, negative-to-unsigned, and 2^53 collision values. Keep DML on its existing type-stable cache path ([Performance]: Wider integer bindings suppress native filters on INT and UNSIGNED keys #29511).
Preserve native BIGINT filtering for prepared
ROUND/TRUNCATEwith zero precision when the text binding is provably an exact integer. Cover direct, scalar-subquery, and projected-marker shapes; retain comparison fallback for fractional, unsafe, NULL, and explicit-CAST inputs ([Performance]: Prepared ROUND numeric text can disable BIGINT filter pushdown #29512).Preserve native DECIMAL filtering when a DOUBLE peer maps to exactly one value at the column scale. Reuse the existing comparison type selection and prepared binding state. Fall back for values between DECIMAL scale points, out-of-range values, nonfinite values, and floating-point collisions at large magnitudes ([Performance]: DOUBLE comparison on DECIMAL column loses block filtering #29510).
Retain that proof through folded scalar expressions and singleton derived-table peers, including ABS and CAST, so their DECIMAL scan or join filters are pushed down. Keep nonconstant or unsafe peers on the existing comparison path ([Performance]: foldable DOUBLE expressions and derived peers lose DECIMAL block filtering #29514).
Apply the same proof to bound DOUBLE parameters inside scalar subqueries, deterministic functions, and singleton derived tables. Reuse the existing prepared value-dependent cache guard and binding-by-position owner, including statements with multiple parameters (Prepared DOUBLE expression peers lose DECIMAL filter pushdown #29515).
Include CAST type descriptors when evaluating safe bound DOUBLE peers, allowing explicit prepared DOUBLE casts in direct, scalar, and derived forms to retain DECIMAL block pruning. Text transport, malformed values, NULL, and floating-point collision cases remain on their existing comparison paths (Explicit prepared DOUBLE casts still disable DECIMAL block pruning #29516).
Reuse the same bound-value proof for complete numeric text passed through an explicit prepared DOUBLE cast. Fold only the proof copy, preserving executable casts and warning behavior; incomplete text and nonunique DOUBLE values retain their fallback (Complete text bindings in explicit DOUBLE cast still miss DECIMAL pruning #29517).
Design: reuse the existing prepared binding, expression-reset, and compile owners. The value-dependent proof uses the existing binding-state cache guard; no parallel cache or execution state machine was added.
Validation
Full UT:
go test -p 1 -mod=readonly ./pkg/sql/plan/function ./pkg/sql/plan ./pkg/sql/compile ./pkg/frontend -count=1passed on the final Explicit prepared DOUBLE casts still disable DECIMAL block pruning #29516 source.Focused race UT across the same four packages passed, including prepared integer comparison, ROUND/TRUNCATE, JSON cast, executable EXPLAIN, and block-filter proof tests.
Incremental SCA: gofmt, go vet, golangci-lint 2.5.0 with Go 1.26; 106 changed Go files / 11 packages against the original fix: restore prepared TPCC filter selectivity with per-binding proof #29462 base, 0 issues.
BVT on the Explicit prepared DOUBLE casts still disable DECIMAL block pruning #29516 final source: [Performance]: foldable DOUBLE expressions and derived peers lose DECIMAL block filtering #29514/Prepared DOUBLE expression peers lose DECIMAL filter pushdown #29515/Explicit prepared DOUBLE casts still disable DECIMAL block pruning #29516 103/103, [Performance]: DOUBLE comparison on DECIMAL column loses block filtering #29510 33/33, [Performance]: Prepared ROUND numeric text can disable BIGINT filter pushdown #29512 44/44, [Performance]: Wider integer bindings suppress native filters on INT and UNSIGNED keys #29511 34/34, [Bug]: Prepared non-integer values on signed BIGINT keys lose block filtering #29506 40/40, [Bug]: EXPLAIN ANALYZE FORCE EXECUTE fails on prepared plan snapshot binding #29509 16/16, [Compatibility]: JSON-to-character CAST unquotes JSON strings instead of serializing them #29471 8/8, [Bug]: Prepared ROUND/TRUNCATE lose runtime numeric domain #29505 15/15, existing [Bug]: Full TPCC suite accumulates long-lived row locks and cascades into lock wait timeouts #29429 71/71.
Black-box [Performance]: Wider integer bindings suppress native filters on INT and UNSIGNED keys #29511 scan QA on a 100,000-row table: safe INT and UNSIGNED BIGINT integer bindings each read 1 block; out-of-range and negative-to-unsigned bindings retained the old fallback and rows. Binary prepared int32/int64 bindings were executed 100 times each; existing DML compile-cache UT was retained.
Black-box [Bug]: Prepared non-integer values on signed BIGINT keys lose block filtering #29506 scan QA on a 100,000-row/13-block table: an integral text binding and native constant each read 1 block under
EXPLAIN ANALYZE FORCE EXECUTE; fractional text conservatively retained the prior scan and result. SQL and binary-protocol bindings, malformed text, BETWEEN/IN, INT32/BIGINT, and the signed 2^53 collision boundary were challenged.Sequential TPCC 10-10, three minutes per version, independently cloned from the same GOOD-readable dataset snapshot: this head 4699.34 tpmC, GOOD predecessor 4842.22 tpmC (head -2.95%), 0 transaction errors on both. A separate same-dataset isolation of pre-[Bug]: Prepared non-integer values on signed BIGINT keys lose block filtering #29506 head versus [Bug]: Prepared non-integer values on signed BIGINT keys lose block filtering #29506 head was 4153.94 versus 4148.88 tpmC (-0.12%), both 0 errors. These short local runs establish no measurable incremental TPCC regression from [Bug]: Prepared non-integer values on signed BIGINT keys lose block filtering #29506; the full head-to-GOOD difference remains visible and is not claimed resolved.
Sequential same-snapshot TPCC 10-10, three minutes per version, isolating [Performance]: Wider integer bindings suppress native filters on INT and UNSIGNED keys #29511: previous head
75521448224449.74 tpmC versus [Performance]: Wider integer bindings suppress native filters on INT and UNSIGNED keys #29511 candidate 5019.17 tpmC (+12.8%), 0 errors both. One short local run does not establish a stable throughput gain but shows no obvious incremental regression.[Performance]: Prepared ROUND numeric text can disable BIGINT filter pushdown #29512 black-box scan QA on 100,000 rows / 13 blocks: safe direct, scalar-subquery, and derived-table bindings read 1 block; fractional input kept the 13-block fallback. Binary-protocol output matched the prior head across string, bytes, float, NULL, malformed, and 2^53 boundary inputs. An explicit SQL CAST kept its prior comparison plan.
Sequential same-snapshot TPCC 10-10, one-minute warmup and three-minute measurement per version, isolating [Performance]: Prepared ROUND numeric text can disable BIGINT filter pushdown #29512: previous head
c0df2e88484121.35 tpmC versus [Performance]: Prepared ROUND numeric text can disable BIGINT filter pushdown #29512 4410.47 tpmC (+7.0%), 0 errors both. This short local run shows no observed incremental regression, not a stable throughput gain.[Performance]: DOUBLE comparison on DECIMAL column loses block filtering #29510 black-box SQL and binary-protocol QA: on 100,000 rows / 13 blocks,
d=CAST(54321 AS DOUBLE)improved from 13 scanned blocks to 1. On the same data, 50 binary prepared float64 lookups took 3.11 s on the prior head versus 0.060 s here. A fractional value between DECIMAL scale points (0.104) retained the old result and fallback time (2.99 s versus 2.92 s for 50 executions). Binary prepared results matched the prior head for safe/fractional/negative/out-of-range/NULL/nonfinite values and the signed 2^53 collision boundary.Sequential same-snapshot TPCC 10-10, one-minute warmup and three-minute measurement per version, isolating [Performance]: DOUBLE comparison on DECIMAL column loses block filtering #29510: prior head
d5dbf378384336.69 tpmC versus [Performance]: DOUBLE comparison on DECIMAL column loses block filtering #29510 4555.56 tpmC (+5.0%), 0 transaction errors both. This short run shows no observed incremental regression; it is not a stable throughput gain.[Performance]: foldable DOUBLE expressions and derived peers lose DECIMAL block filtering #29514 black-box QA on 100,000 rows / 13 blocks: safe scalar subquery, ABS(CAST(...)), derived join, and derived ABS peer shapes changed from 13 scanned blocks to 1. Fractional peers retained the 13-block fallback; row counts remained the same, including the signed 2^53 collision case. DOUBLE(M,D) peers are conservatively excluded from the proof.
Sequential same-snapshot TPCC 10-10, one-minute warmup and three-minute measurement per version, isolating [Performance]: foldable DOUBLE expressions and derived peers lose DECIMAL block filtering #29514: prior head
882edd215c4739.25 tpmC versus [Performance]: foldable DOUBLE expressions and derived peers lose DECIMAL block filtering #29514 4599.64 tpmC (-2.95%), 0 transaction errors both. One short run cannot distinguish normal variation from a small regression.Prepared DOUBLE expression peers lose DECIMAL filter pushdown #29515 black-box binary prepared QA on 100,000 DECIMAL rows / 13 blocks: 30 safe lookups went from 1.36–1.58 s on the prior head to 20–45 ms for scalar, ABS, scalar ABS, derived join, and derived ABS shapes. The non-representable
0.104peer retained its existing fallback and result. A 50-case matrix across float, text, NULL, negative, nonfinite, CAST, and repeated safe/unsafe bindings matched the prior head exactly. SQL PREPARE BVT also covers a second parameter in a different execution slot and the 2^53 collision boundary.Sequential same-snapshot TPCC 10-10, one-minute warmup and three-minute measurement per version, isolating Prepared DOUBLE expression peers lose DECIMAL filter pushdown #29515: old-first pairing was prior head
79a76763b24174.14 versus Prepared DOUBLE expression peers lose DECIMAL filter pushdown #29515 4061.56 tpmC (-2.70%); new-first pairing was Prepared DOUBLE expression peers lose DECIMAL filter pushdown #29515 4209.81 versus prior head 4115.00 tpmC (+2.30%). Zero transaction errors in all four measurements. Across those runs, prior averaged 4144.57 and Prepared DOUBLE expression peers lose DECIMAL filter pushdown #29515 averaged 4135.69 tpmC (-0.21%); this local duration cannot resolve a difference that small.Explicit prepared DOUBLE casts still disable DECIMAL block pruning #29516 black-box SQL PREPARE: safe explicit CAST, scalar CAST, and derived CAST each changed from 13 scanned blocks to 1 on 100,000 DECIMAL rows; the signed 2^53 collision still returns both matching rows. A 50-case binary prepared matrix matched the prior head exactly; malformed text
54321junkandabckept the same result and warning 1292.Sequential same-snapshot TPCC 10-10, one-minute warmup and three-minute measurement per version, isolating Explicit prepared DOUBLE casts still disable DECIMAL block pruning #29516: old-first pairing was prior head
e0082f4be04111.62 versus Explicit prepared DOUBLE casts still disable DECIMAL block pruning #29516 3957.17 tpmC (-3.76%); new-first pairing was Explicit prepared DOUBLE casts still disable DECIMAL block pruning #29516 4034.92 versus prior head 4628.58 tpmC (-12.83%). All four measurements had 0 transaction errors. The prior head itself varied 12.6% across the two runs. Inspection of every SQL string in the bundled TPCC client found no explicit CAST or DECIMAL-column filter, so these transactions cannot enter the Explicit prepared DOUBLE casts still disable DECIMAL block pruning #29516 rewrite branch; this is an inference from the workload SQL, not a claim that the observed throughput gap has been explained. Fixed-query QA demonstrates the affected DECIMAL filter improving from 13 scanned blocks to 1 with unchanged results.Complete text bindings in explicit DOUBLE cast still miss DECIMAL pruning #29517 binary prepared performance, 30 executions on 100,000 DECIMAL rows / 13 blocks: text
"54321"changed from 1.56 s to 56 ms and from 13 scanned blocks to 1. Partial numeric"54321junk"remained on its old fallback (1.36 s before, 1.39 s after); nonrepresentable"0.104"still scans 13 blocks. A 10-value SQL PREPARE matrix including whitespace, sign, exponent, malformed text, overflow and NULL produced byte-identical results and warnings against the prior head. BVT covers direct, scalar, derived, ABS, warning 1292 and 2^53 collision behavior.Final local Complete text bindings in explicit DOUBLE cast still miss DECIMAL pruning #29517 validation: full UT in four related packages passed; incremental SCA passed with 106 Go files / 11 packages and 0 issues; all 10 affected BVT files passed (443 statements total, including Complete text bindings in explicit DOUBLE cast still miss DECIMAL pruning #29517 140/140). Focused race UT across the four related packages passed.
Full PR head
ff664d1d75versus pre-fix: restore prepared TPCC filter selectivity with per-binding proof #29462 GOOD1575e70c3d: sequential TPCC 10-10, two same-snapshot pairs in opposite order, each with 2-minute warmup and 2-minute measurement. GOOD measured 4665.92 and 3858.81 tpmC; PR measured 5309.81 and 4677.13 tpmC; all four runs had 0 transaction errors. Means were 4262.36 GOOD and 4993.47 PR (+17.1%), but within-version variation was 12.7–18.9%. The host showed sustained I/O pressure during this test, so these short results are directional and do not establish a stable TPCC gain. This compares the cumulative PR with GOOD; it does not isolate Complete text bindings in explicit DOUBLE cast still miss DECIMAL pruning #29517.Follow-up QA
The fix is scoped to comparisons where a unique DECIMAL value can be proven. Full sysbench/TPC-H/ClickBench workload performance remains subject to broader QA. Additional prepared expression and nonconstant peer shapes will continue to be challenged.
CI follow-up: assignment and JSON conversion boundaries
Latest repair commit:
bc7c123ebd.BindContext.assignmentIgnorefield and its propagation.Current local evidence:
TestPreparedSpecializedDomains, with race enabled: PASS.-nresult comparison mode.Sequential TPCC 10-10 for
bc7c123ebdversus GOOD1575e70c3dcompleted: 4498.76 versus 4291.20 tpmC (+4.84%), zero errors, 2-minute warmup and 2-minute measurement on independent copies of the same initial snapshot. One pair does not establish a stable gain. That head later failed Ubuntu UT at issue29400's 2-second DROP deadline; Coverage separately exhausted retries on GitHub API HTTP 503.Numeric predicate follow-up and CI test hardening
Latest repair:
811ab25f9c.Validation on this source:
Sequential TPCC 10-10 for
811ab25f9cversus GOOD1575e70c3dcompleted: 4119.76 versus 3897.51 tpmC (+5.70%), 9297.80 versus 8718.62 tpmTOTAL, zero transaction errors on both, 2-minute warmup plus 2-minute measurement per version, candidate then GOOD with independent copies of the same initial snapshot. Other services, build and lint workloads were active on the shared machine. Sampled system I/O pressure was substantial; collection starts during the candidate measurement. These numbers are observations, not evidence of a stable gain or an isolated regression result. The deterministic block-count counterexamples above establish the targeted pruning improvement.Fresh required CI is still running. The user requested not to wait for CI; local validation does not claim remote CI green or universal workload coverage.